An affinity matrix is a generic matrix that measures pairwise relationships between points, indicating how close, or similar, two points are in a space. Affinity matrices are widely used in computer vision problems, representing a weighted graph that regards each pixel as a node and connects each pair of pixels by an edge. The weight (affinity value) on an edge should reflect the pairwise similarity with respect to a task. For example, for low-level vision tasks such as image filtering, the affinity values should reveal the low-level coherence of color and texture; for mid to high-level vision tasks such as image matting and segmentation, the affinity values should reveal the semantic-level pairwise similarities. Most techniques explicitly or implicitly assume a measurement or a similarity structure over the space of configurations. The success of a technique depends heavily on the assumptions made to construct the affinity matrices, which are generally constructed manually. There is a need for addressing these issues and/or other issues associated with the prior art.